Skip to content

feat(webhooks): verify_and_parse_* API for compressed payloads (CHA-3071)#211

Open
nijeesh-stream wants to merge 7 commits into
masterfrom
nijeeshjoshy/cha-3071-compress-webhook-payloads
Open

feat(webhooks): verify_and_parse_* API for compressed payloads (CHA-3071)#211
nijeesh-stream wants to merge 7 commits into
masterfrom
nijeeshjoshy/cha-3071-compress-webhook-payloads

Conversation

@nijeesh-stream
Copy link
Copy Markdown
Contributor

@nijeesh-stream nijeesh-stream commented May 7, 2026

Summary

Adds first-class support for gzip-compressed webhook payloads (HTTP webhooks, SQS, SNS) and exposes a stable verify_and_parse_* API that mirrors the cross-SDK contract published in Webhooks Overview.

New public API (StreamChat::Webhook)

Module-level primitives:

  • gunzip_payload(body) -> String — gzip-magic-byte detection, no-op when not compressed
  • decode_sqs_payload(body) — base64 decode then gunzip-if-magic
  • decode_sns_payload(notification_body) — JSON-parse the SNS HTTP notification envelope, extract the inner Message, then run the SQS pipeline. Falls through to a pre-extracted Message string when the input is not a JSON envelope
  • verify_signature(body, signature, secret) -> bool — HMAC-SHA256 over the uncompressed body, with a constant-time comparison (matters for the HTTP webhook path where the X-Signature header is exposed publicly; SQS / SNS deliveries arrive over AWS-internal transports where timing-attack resistance is not strictly required)
  • parse_event(payload) -> Hash — JSON → Hash

Module-level composites (return Hash):

  • verify_and_parse_webhook(body, signature, secret) -> Hash
  • verify_and_parse_sqs(body, signature, secret) -> Hash
  • verify_and_parse_sns(body, signature, secret) -> Hash

The StreamChat::Client instance also exposes verify_and_parse_webhook / verify_and_parse_sqs / verify_and_parse_sns that use the configured @api_secret.

Typed Event objects will land in Ruby in a follow-up release. Until then the helpers return the parsed JSON as a Hash.

Backwards compatibility

Client#verify_webhook is preserved and now delegates to StreamChat::Webhook.verify_signature. The experimental decompress_webhook_body and verify_and_decode_webhook surfaces are removed (they were never released).

Tests

spec/webhook_compression_spec.rb covers plain / gzip / base64 / base64+gzip payloads, signature mismatches, malformed bytes, and JSON parsing into Hash. Linked Linear ticket: CHA-3071.

Golden test fixtures (Tommaso)

Added shared reference fixtures to the test suite so future SDKs can sanity-check decoders against the same payloads:

aGVsbG93b3JsZA==                          -> helloworld   (base64)
H4sIAGrYAWoAA8tIzcnJL88vykkBAK0g6/kKAAAA -> helloworld   (base64 + gzip)

Test plan

  • bundle exec rspec spec/webhook_compression_spec.rb — 28 examples, 0 failures
  • bundle exec rubocop — clean on touched files

… (CHA-3071)

Adds two new helpers on `StreamChat::Client` so customers can ingest
gzip-compressed webhook payloads (and base64-wrapped SQS / SNS firehose
envelopes) without wiring `Zlib`, `Base64`, and `OpenSSL::HMAC`
together themselves:

* `decompress_webhook_body(body, content_encoding = nil, payload_encoding = nil)`
  - the primitive decoder, no signature check
* `verify_and_decode_webhook(body, x_signature, content_encoding = nil, payload_encoding = nil)`
  - decodes and timing-safely verifies the X-Signature HMAC against the
    uncompressed JSON, raising `StreamChat::WebhookSignatureError` on
    any failure (decode, decompress, or signature mismatch).

`nil` / empty for either encoding is a no-op, so the same handler keeps
working whether or not compression is enabled at the dashboard.

The existing `verify_webhook(body, x_signature)` boolean helper is left
untouched for backward compatibility.

Helper logic lives in `lib/stream-chat/webhook.rb` so the decode
primitives can be exercised independently. New tests in
`spec/webhook_compression_spec.rb` cover plain / gzip / base64 / base64
+ gzip round-trips, case-insensitive encoding values, every documented
unsupported encoding, malformed gzip / base64 input, and signature
mismatch scenarios (including signatures wrongly computed over the
compressed or wrapped bytes).

Docs: new "Compressed webhook bodies" section in the webhooks overview
with a Rails example and an SQS / SNS subsection that passes
`payload_encoding: 'base64'`.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replaces verify_and_decode_webhook / decompress_webhook_body with the
cross-SDK contract documented at
https://getstream.io/chat/docs/node/webhooks_overview/.

Module-level helpers in StreamChat::Webhook:

  Primitives:
    ungzip_payload        - gzip magic-byte detection + inflate
    decode_sqs_payload    - base64 then ungzip-if-magic
    decode_sns_payload    - alias for decode_sqs_payload
    verify_signature      - constant-time HMAC-SHA256 comparison
                            (parameter order matches the cross-SDK spec)
    parse_event           - JSON -> Hash (typed event lands later)

  Composite (return parsed event Hash):
    verify_and_parse_webhook
    verify_and_parse_sqs
    verify_and_parse_sns

The composite functions auto-detect compression from body bytes, so
the same handler stays correct whether or not Stream is currently
compressing payloads, and behind middleware that auto-decompresses.

Client#verify_and_parse_{webhook,sqs,sns} mirror the three composite
helpers with the api_secret pulled from the client.

The legacy Client#verify_webhook(body, signature) -> bool helper is
kept for backward compatibility (now delegates to verify_signature).

Co-authored-by: Cursor <cursoragent@cursor.com>
@nijeesh-stream nijeesh-stream changed the title feat(webhooks): add verify_and_decode_webhook for compressed payloads feat(webhooks): verify_and_parse_* API for compressed payloads (CHA-3071) May 8, 2026
nijeesh-stream and others added 3 commits May 8, 2026 16:53
RFC 1952 defines the gzip magic number as the two-byte sequence
1F 8B; the third byte (CM) is informational and not part of the
identifier. Trim the magic check from three bytes to two to match
the spec and stay consistent with the reference implementations
in the public docs.

Co-authored-by: Cursor <cursoragent@cursor.com>
The previous parent (`StreamAPIException`) declares a non-nilable
`Faraday::Response` and an HTTP-shaped `initialize`. A local webhook
verification failure has no response, so the subclass was forced to
skip `super`, fake `@error_code`/`@error_message`/`@json_response`,
and call `StandardError#initialize` via `bind_call` to populate the
message - while still exposing a `response` reader that violates its
own type signature by returning nil.

Inherit directly from `StandardError` instead. Drops the LSP
violation, the `Lint/MissingSuper` disable, and three irrelevant
accessors. No caller depended on the prior parent: the class was
introduced in this PR and is only used by the webhook helpers.

CHA-3071

Co-authored-by: Cursor <cursoragent@cursor.com>
The Ruby code samples still referenced an earlier draft of the API
(`decompress_webhook_body`, `verify_and_decode_webhook`, with
`content_encoding` / `payload_encoding` arguments returning raw JSON
bytes). The shipped API renamed those to `verify_and_parse_*`,
removed the encoding arguments (gzip is detected from the body
bytes per RFC 1952), and returns a parsed `Hash` rather than bytes -
so the previous snippets would have failed with NoMethodError, and
the subsequent `JSON.parse(json_bytes)` would have raised TypeError
on the returned Hash.

Replace both snippets with the current API, add a pointer to the
module-level primitives, and note that the legacy boolean
`verify_webhook` remains for callers that don't need compression
support.

CHA-3071

Co-authored-by: Cursor <cursoragent@cursor.com>
Copy link
Copy Markdown
Contributor

@mogita mogita left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-SDK review pass for CHA-3071. Two inline comments — see below.

Comment thread lib/stream-chat/webhook.rb Outdated
Comment thread spec/webhook_compression_spec.rb
decode_sns_payload now JSON-parses the SNS HTTP notification envelope
({"Type":"Notification","Message":"..."}) and extracts the inner
Message field before running the SQS pipeline. Falls through to the
pre-extracted Message string when the input is not a JSON envelope so
existing call sites keep working.

Spec adds a realistic SNS HTTP notification body fixture and exercises
both the new envelope path and the existing pre-extracted Message path.

Co-authored-by: Cursor <cursoragent@cursor.com>
@mogita
Copy link
Copy Markdown
Contributor

mogita commented May 11, 2026

Cross-SDK coordination: unifying webhook exception types

After the review pass across all 6 SDKs in this rollout and team discussion, we're consolidating the new webhook exception strategy to a single unified exception class rather than the split (signature vs parse exceptions) being introduced in this PR.

The Webhook Handling Spec on Notion (CHA-2961) has been revised to reflect this — §5.2 / §5.3 / §7 now specify a single class.

Why unified: From a customer's perspective, all failure modes — signature mismatch, gzip decompression failure, base64 decode failure, SNS envelope failure, JSON parse failure, missing schema field — terminate at the same rescue block in customer code. A signature/parse split adds structural complexity without changing customer behavior. Customers who want to filter security logs for signature mismatches specifically can do so via exception message text or Exception#cause chain.

Class name family: InvalidWebhookError — "Invalid" covers all failure modes accurately, consistent with Ruby's stdlib naming patterns (ArgumentError, etc., where the prefix names the category of invalid input).

Per-SDK naming across the rollout:

SDK Class name
JS InvalidWebhookError (extends Error)
Python InvalidWebhookError
Go sentinel ErrInvalidWebhook + struct InvalidWebhookError
Java InvalidWebhookException (extends existing StreamException)
PHP InvalidWebhookException (extends existing StreamException)
Ruby StreamChat::InvalidWebhookError (extends StandardError)
.NET StreamInvalidWebhookException (extends StreamBaseException)

Asks for this PR:

  1. Rename WebhookSignatureErrorStreamChat::InvalidWebhookError
  2. Wrap all failure paths into this single type — signature mismatch, gzip failure, base64 failure, SNS envelope failure, JSON parse failure, missing type/schema failure (including the currently-leaking JSON::ParserError — wrap it)
  3. Attach a human-readable message identifying which failure mode fired (e.g. "signature mismatch", "invalid base64", "missing type field") so customers can filter on message content
  4. Legacy Client#verify_webhook (returning Boolean) stays unchanged — back-compat preserved
  5. Update specs to assert against the new exception name; for mode-specific tests, also assert on message-content substrings

This same comment is being posted on all 6 SDK PRs (JS / Go / Ruby / PHP / Java / .NET) for coordination. Happy to discuss naming or scope tradeoffs.

…den fixtures (CHA-3071)

Per Tommaso's suggestion, align the gzip helper with the GNU `gunzip`
command name. The function was added in this PR and not yet released,
so this is a straight rename with no back-compat alias.

Adds Tommaso's reference fixtures to the test suite as named cases so
future SDKs can sanity-check against the same payloads:

  aGVsbG93b3JsZA==                          -> helloworld   (base64)
  H4sIAGrYAWoAA8tIzcnJL88vykkBAK0g6/kKAAAA -> helloworld   (base64+gzip)

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants